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Abstract 

Assessment practices and requirements are very broad and vary widely among academic programs 
and from one institution to the other. Consequently, we noticed, in the recent years, increased 
volumes of research and interest geared into the assessment process and procedures in various 
disciplines in higher education. In this paper, we present and explain a model for long term 
assessment and a set of robust tools and techniques within the framework of process steps, team 
work, and task-driven process management. Using this presented assessment methodology, we have 
been successful in our accreditation efforts, and improved the quality of our programs. This model can 
be used for long-term assessment with several years of task scheduling and assessment timeline. We 
share our views and thoughts in the form of lessons learned and best practices so as to streamline the 
process of assessment and simplify its procedures and steps. 

Keywords: Assessment, Assessment model, Assessment tools. Information Systems programs, 
Accreditation 


1. INTRODUCTION 

Motivation: Assessment practices and 

requirements are very broad and can be 
interpreted and applied in many ways. In this 


work, the main motivation is to tackle an 
assessment process and present a well specified 
assessment model and set of tools with the 
framework of process steps, team work, and 
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project based task. Moreover, this work can be 
viewed as a way to share and disseminate our 
work practices, findings, and lessons learned in 
an assessment task. 

Background: For an educational accreditation 
purpose, a certain form of assessment is 
typically mandated by a national or regional 
accreditation agency such as ABET, AACSB, and 
SACS, with the main responsibility of 
maintaining the standards for degree 
confirmation. An assessment process can be 
viewed as a simple and direct evaluation of an 
academic program or discipline in an educational 
institution. Assessment can be accomplished at 
various levels. Typically three levels of 
assessment can be distinguished: 

• Institution-level assessment, 

• School-level assessment, e.g. school of 
business, or school of education 

• Program-level assessment, e.g. 
information systems program or 
accounting program. 

Program-level assessment is the focus of this 
paper. 

Reasons for Assessment: Reasons for 
assessment can be grouped into three major 
categories: (1) to satisfy external accreditation 
requirements at various levels: university, 
school and program; (2) to satisfy internal 
requirements of the university, such as periodic 
program reviews, etc.; and (3) to utilize the 
results internally to improve the programs or for 
recruiting and marketing purposes. 

Goals of This Paper: This paper presents and 
explains a set of robust and compressive 
assessment guidelines for computing and 
information systems fields. We designed and 
implemented a comprehensive assessment 
methodology for two computing programs. We 
started with the mission statement and 
streamlined the main objectives of the 
programs. The method includes a 
comprehensive and solid set of measurable goals 
and outcomes. The results of applying this 
assessment methodology are then taken into the 
last phase which is known as 'closing the loop'. 
In the closing the loop phase, we take the 
assessment results and apply the 
recommendations to improve the quality of the 
programs. We have been using this presented 
assessment methodology for several years and it 
has helped us to improve the quality of our 
programs. Moreover, this assessment method 
has helped and simplified the accreditation 
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process of two computing programs by ABET 
under the IS and CS curriculum guidelines. 

2. LITERATURE REVIEW 

Faculty who recognize the advantages of an 
accredited program are familiar with curriculum 
models and accreditation requirements. Landry, 
et al. (2009; 2006) discuss the Information 
Systems (IS) 2002 model curriculum and how 
150 learning units are mapped into 6 IS core 
areas. The model curriculum is a result of a 
collaborative effort that describes the 
characteristics of the IS profession. 

Hilton, et al. (2003; 2004) conduct a comparison 
of the school-level Association to Advance 
Collegiate Schools of Business (AACSB) and 
program-level ABET/CAC accreditation 
standards. They find AACSB and ABET/CAC 
accreditation standards to be generally 
compatible. Based on a survey of IS program 
leaders in business schools, understanding of 
potential benefits of accreditation were quite 
low. Challa, et al. (2005) find that much of the 
requirements of ABET, including assessment, is 
applicable to IS programs. 

Nicolai (2004) addresses the dilemma of how a 
particular curriculum is positioned into an 
accreditation model. She concludes that "IS 
expects database students to achieve a higher 
level of learning (application) and IT expects 
database students to achieve the first level of 
learning (understanding)." 

Sun (2003) and Kortsarts et al. (2009) discuss 
the technical and personal skills that need to be 
mastered in order to be an effective IT person. 
Necessary skills include: helpdesk skills, 
programming and optimizing code, systems 
administration, security, systems integration, 
database, web mastering, knowledge of disaster 
recovery procedures, and business planning. 
Such a person would also possess personal 
skills: creativity to know whether a thing is 
possible, ways to work around problems, 
organization skills, interpersonal skills, the 
ability to explain complexities in simple terms, to 
link components together, to see where future 
growth can happen, to work effectively on a 
team, and the spirit and practice of cooperation. 
The assessment of such skill mastery is, thus, 
critical to an IT program. 

3. A CONCEPTUAL MODEL OF ASSESSMENT 

Adapting the basic components of assessment 
from the ABET Assessment for Quality Assurance 
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Model (ABET, 2010), we propose a conceptual 
model of assessment (see Figure 1), which 
could be used to prepare educational 
assessment in general. The conceptual model 
consists of three parts: 

institutional/school/program level's guidance 
components, evaluation components, and 
feedback. The guidance components are related 
to the direction of institutional/school/program, 
which include mission, objectives, and 
outcomes. A mission is a broad and long-term 
vision of an institution/school/program. There 
will be objectives, outcomes, and strategies used 
to achieve the mission, but the mission is the 
eminent and most important aim to be 
accomplished. Objectives, on the other hand, 
are broad statements that describe the career 
and professional accomplishments that the 
program is preparing graduates to achieve. 
Outcomes are statements that describe what 
students are expected to know and are able to 
do by the time of graduation (Vlasceanu, 
Grunberg, & Parlea, 2007). If students have 
achieved these outcomes, it is anticipated that 
they will be able to achieve the educational 
objectives after graduation. Appendix 1 shows 
an example of program level mission statement, 
educational objectives and outcomes. The 
evaluation components include performance 
measurement criteria of the guidance 
components, assessment of performance, and 
interpretation of the results of assessment. 
While the guidance components are about 
"where to go," the evaluation components are 
related to analysis mechanisms to answer 
"where do we stand." 

Performance criteria are specific and measurable 
statements identifying the performance(s) 
required to meet outcomes (Prados, Peterson, & 
Lattuca, 2005). 

These should be high level measurable 
statements that represent the knowledge, skills, 
attitudes or behavior students should be able to 
demonstrate by the time of graduation. 
Assessment is related to the processes that 
identify, collect, use and prepare data that can 
be used to directly or indirectly evaluate 
performance (i.e., achievement). Interpretation 
is the process that is used to interpret the 
meaning of the assessment results and provide 
recommendations. The feedback process is 
critical to creating and maintaining a systematic 
quality assurance system. When successfully 
implemented, all elements of the quality 
assurance process interact with one another 
(ABET, 2010). This model can easily be mapped 
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to the assessment requirements of accreditation 
bodies such as ABET (2010), AACSB (2010), and 
SACS (2010), as well as the internal needs and 
framework for program improvement. 



Accreditation/course committee level 


Figure 1: Conceptual Model of Assessment 

4. ASSESSMENT MODEL IMPLEMENTATION 

We followed this model to prepare for both ABET 
accreditation and internal program 
improvement. A committee of five dedicated 
faculty members were selected for the 
assessment committee. This committee met 
regularly to spearhead the assessment cycle. 

Assessment Methods: The assessment 
committee identified several methods in which 
assessment of outcomes could be done. Some of 
the methods identified were indirect methods of 
assessment, while others were direct methods 
(see Appendix 2 for an example). Indirect 
methods were easy to implement and less time 
consuming. The best example of indirect 
methods is the exit student survey on how well 
they think course objectives are satisfied. 

The results of these were not as convincing as 
they were more of an opinion rather than a fact. 
Nonetheless, they can be useful to effectively 
identify issues that need to be improved. Direct 
methods, on the other hand, were much more 
time consuming to the instructor; however these 
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results were more relevant and accurate. The 
committee identified and adapted 11 assessment 
methods, out of which 9 were direct methods of 
assessment. 

It was critical that the faculty did not get 
overloaded with assessment. Overloading the 
faculty would have been a recipe for failure. It 
was imperative that assessment methods were 
assigned evenly. Therefore, some of the direct 
methods were identified to be prepared and 
judged by the course instructor, while others 
were assigned to be evaluated by a committee 
of faculty members or the course committees 
and still others were assigned to be assessed by 
industrial advisors. 

Mapping Courses to Outcomes: It is 

necessary to find out which courses would 
satisfy the different outcomes for each program 
objective. The course committees for each class 
identified course goals for each course, and with 
these course goals, the assessment committee 
identified which courses mapped to which 
outcomes (see Appendix 3). 

Assessment Timeline: After learning outcomes 
and multiple assessment methods were decided, 
it was imperative that the assessment cycle was 
achievable, so that assessment did not fall 
through the cracks. It was the view of the 
committee that a good timeline that did not 
overly tax any one course or one person much 
would be a more practical assessment cycle. The 
timeline took into consideration each objective 
for the program. Every outcome in each 
objective was assessed once every 3 years. In 
this way, most classes were assessed once 
every 3 years, or every 2 years at most. This 
seemed like a very achievable plan with very 
minimum impact on workload, which is a 
common concern among faculty (Hogan, 
Harrison, & Schulze, 2002). 

Closing the Loop: Assessment on its own 
would have no impact on the program. The 
resultant recommendations and outcomes from 
the assessment would eventually make for a 
better program. As feedback mechanisms of 
quality assurance, the recommendations that 
are applied to the assessment results not only 
improve the program, but also give us 
information about the quality of classes, the 
quality and ability of our students, and shows us 
where we have to improve. 

In closing the loop, we realized that some 
classes were overloaded with material. In some 
instances, it was necessary to add more material 
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into classes, while some classes required no 
changes. Other categories of recommendations 
we had implemented include changes in 
program and course outcomes, changes in 
performance criteria and assessment tools, 
increases in course support and changes in 
instructors. 

5. DISCUSSION AND LESSONS LEARNED 

In this section, we present a discussion of our 
views and thoughts on the assessment. We also 
discuss the lessons learned in this work. These 
views, thoughts and lessons learned are 
summarized in the form of three best practices 
as follows. 

Best Practices: Formation of a program 
accreditation and assessment committee 

In the past, our ABET accreditation effort was 
spearheaded by one or two individual faculty 
members, usually the program chairs. This 
resulted in uneven faculty participation and 
missed tasks. Despite best efforts and successful 
accreditations, the experience was less than 
fulfilling for all those involved. There was not 
sufficient discussion among faculty members to 
recommend and implement comprehensive 
changes to improve the programs. Efforts were 
focused only on issues of perceived weaknesses 
related to accreditation. Furthermore, the 
concentration of work created stress for the lead 
persons. 

However, it is also not realistic to manage 
accreditation preparation through the entire 
faculty body. We tried to discuss nuanced 
accreditation issues during full faculty meetings 
in the past which usually ended inefficiently as 
faculty with different levels of understanding 
tended to over discuss unimportant issues and 
details. The uneven level of contributions during 
and after the meetings also discouraged faculty 
participation. 

In the latest ABET accreditation cycle, we 
formed a committee of five devoted faculty 
members to lead the effort for both accreditation 
and assessment. This turned out to be a suitable 
size for gathering ideas and actually executing 
the preparation plan. Every member was active. 
As the committee successfully resolved tasks 
effectively, a culture of teamwork established. 
The resulting collaboration continued beyond 
accreditation and assessment, resulting in 
resolving other program matters and 
publications of papers. Merging the accreditation 
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and the broader assessment efforts also 
reinforced each other. 

Best Practices: Adoption of a management 
process for accreditation and assessment 

Accreditation and assessment involve many 
concurrent tasks to prepare a large collection of 
documents. These tasks need to be identified, 
refined and specified. Solutions to these tasks 
need to be designed and implemented (Mayes & 
Bennett, 2005). Leaders and supporters of tasks 
and deadlines need to be established and 
followed through. Many documents need to 
evolve in time and may also have variations to 
satisfy different needs. Furthermore, documents 
are updated and accessed by many different 
groups of users: faculty members, supporting 
staff, adjunct faculty, course committees, etc. 
Thus, in a sense, accreditation and assessment 
can be regarded as a project with many 
similarities with software development projects: 
risk management, version control, feature 
completeness, etc. 

As a result, an early task our accreditation 
committee undertook was to adopt a reasonable 
project management process. On one hand, we 
needed a process to ensure the systematic 
identification and completion of needed tasks. 
On the other hand, the process needed to be 
informal enough to let innovative ideas flow 
freely. 

As information systems and computer science 
faculty members, we borrowed ideas from 
Rational Unified Process (RUP) (Kruchten, 2003) 
and Scrum Development (Wikipedia, 2010). RUP 
is a leading iterative software development 
framework and Scrum is "an iterative, 
incremental framework for project management 
and agile software development" (Wikipedia, 
2010). Ideas we borrowed from them are 
iterations of task management until completion, 
frequent and systematic status updates, change 
control, continuous quality verification, and 
heightened communications through frequent 
meetings. 

The process we eventually adopted was to hold 
weekly meetings. All documents developed 
during the week were captured in a dedicated 
work area folder which also serves as an archive 
and version control. A progress file, simply in 
Microsoft Word format, documents every task, 
its leader and steps remaining to be done for the 
task. The urgency and progress status of each 
task is color coded. Each task was re-visited 
each week to check its progress with possible re¬ 
examinations of their goals, design and 
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implementations. This ensures that tasks are 
completed effectively within deadlines and that 
no task was missed. The longitudinal sequence 
of progress files also provides a good history of 
progress. 

We were cautious to identify tasks that were 
best resolved during the meeting and they were 
worked upon immediately. For example, the 
assessment committee refined the wordings of 
updated program objectives during the 
meetings. This provided quick consensus so that 
the objectives could be presented to the full 
faculty body for approval rapidly. On the other 
hand, there were many tasks that could be 
accomplished individually after the meeting. 

We would have used project management 
software which provides aids using a more 
formal project management process. However, 
since the key members met frequently in 
person, we found that our informal approach 
incurred the least overhead while keeping 
communications of ideas open. 

Best Practices: Use of technology when 
appropriate 

We used technology to aid the assessment 
process only when the benefit justified the 
overhead. We used an Intranet to provide easy 
access to the myriad of documents we created. 
There were sections to host documents that 
were relatively stable and areas for documents 
that were more volatile, requiring rapid changes. 
We developed a Web database application to 
hold the exit surveys of all undergraduate 
courses. The application also allows members of 
the course committees to enter their 
recommendations, which were then collected, 
discussed and approved. We did not use any 
particular collaborative tool for developing 
documents. Instead, the committee worked 
together to finalize versions created by 
individual members during our meetings. Using 
a real-time collaborative tool, such as 
GoogleWave (2010), is an experiment we will 
pursue in the future. 

6. CONCLUSION AND FUTURE DIRECTIONS 

In higher education institutions, the assessment 
process is a crucial task that can benefit many 
stakeholders. Assessment can be a very broad 
process with no fixed procedure or methodology 
mandated. In the information technology 
disciplines, however, there are certain rules and 
actions that are necessary to accomplish a 
reasonable assessment. In this paper, we 
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presented a process model and some tools for 
assessment for information technology 
programs. 

The future direction in this work is twofold: (1) 
Unifying the terminology and language of the 
assessment. The definitions of the terms for 
assessment may lead to different notions in 
different contexts. Standardized assessment 
language and terminology will lead to simplifying 
operations that build upon assessment, like 
accreditation. (2) Relating model curriculums 
and accreditation requirements for specific 
disciplines with assessment models. This aids in 
using a holistic model to satisfy varying 
assessment goals. With the entire faculty 
participating in the assessment process, it was a 
very positive eye-opener for our program, and 
assessment was definitely a constructive 
addition to our program. 
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8. APPENDICES 

Appendix 1: Program Level Mission Statement (Example) 


The mission of the Computer Information Systems program is to prepare students for technical, 
administrative and management careers in the analysis, design, implementation, maintenance, 
support, operation and management of computer information systems. 


Objectives and Outcomes 

Objective #1: Computer Information Systems graduates will be competent in the fundamentals of 
information systems, computing, and mathematics. 

Outcome 1: Students can present the key concepts and principles of computer and information 
systems. 

Outcome 2: Students will be able to effectively solve computing problems using an appropriate 
programming language, data structures and algorithms. 

Outcome 3: Students can use mathematical concepts in the analysis and design of information 
systems. 


CIS Objective #2: Computer Information Systems graduates will understand the role of IS and be 
able to work effectively within information systems environments. 

Outcome 2.1: Student will be able to identify significant opportunities and problems in information 
systems. 

Outcome 2.2: Students will be able to understand the role of information systems in helping 
individuals and groups make decisions efficiently and effectively. 

Outcome 2.3: Students will be able to evaluate the role of information systems in solving 
significant business problems. 
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Appendix 2: General Assessment Tools (Example) 


1. Examination Analysis fEAl: direct method 

a. Instructors map examination questions to specific performance indicators. 

b. Curriculum committee and instructors decide whether these indicators are satisfied or 
not. 

c. Curriculum committee and instructors make recommendations. 

d. Curriculum committee reviews the assessment results and recommendations. 


2. Assignment Analysis [AA]: direct method (including homework, programming and paper 
assignments) 

a. Instructors select assignments that map to specific performance indicators. 

b. Curriculum committees assess the assignment to decide whether these indicators are 
satisfied or not. 

c. Curriculum committees make recommendations. 


3. Portfolio Analysis fPA“|: direct method 

a. Every faculty member takes turn to serve in portfolio analysis. 

b. A selected group of faculty members assesses specific performance indicators by filling 
out an assessment rubric. 

c. The collected rubric assessment is used to decide whether these indicators are satisfied 
or not. 

d. The group of faculty members makes recommendations. 
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Appendix 3: Mapping of Objectives, Outcomes and Performance Indicators to Course-Level 
Assessment (Example) 

Objectives #1: Computer Information Systems graduates will be competent in the fundamentals of 
information systems, computing, and mathematics. 

Outcome 1.1: Students can present the key concepts and principles of computer and information 
systems. 


Performance 

Indicators 

Strategies 

Assessmen 
t Methods 

Source of 
Assessmen 
t 

Time of 
data 

collection 

Assessment 

Coordinator 

Evaluation 
of Results 

1.1.1 Identify key 
concepts and 
principles of 
information 
systems 

CINF 3231 
CINF 4234 

EA or AA, 

ES 

CINF 3231 

Fall 

Instructor A 

Course 

Curriculum 

Committee 

1.1.2 Evaluate 
the role of 
information 
systems in 
today's 
competitive 
business 
environment 

CINF 3231 
CINF 4234 

EA or AA, 

ES 

CINF 3231 

Fall 

Instructor A 

Course 

Curriculum 

Committee 

1.1.3 Demonstrat 
e the 

understanding of 
the importance of 
information 
systems 

CINF 3231 
CINF 4234 

EA, ES 

CINF 3231 

Fall 

Instructor B 

Course 

Curriculum 

Committee 

1.1.4 Understand 

fundamental 

relationship 

between 

hardware and 

software 

CINF 3231 
CSCI3331 

EA or AA, 

ES 

CSCI3331 

Fall 

Instructor B 

Course 

Curriculum 

Committee 
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